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Overview of Application IDs and 
Fingerprints 



Background of the 4 generations of 
AppIDs+Fingerprints 

Examples of how they are used for target 
development SIGDEV 
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tag given to a session to help describe 
what application is being seen in the traffic 



Examples: 

mail/webmail/yahoo indicates that the traffic was 
Yahoo Webmail 

chat/msn_messenger indicates the traffic was 
MSN Messenger 

http/get indicates that the traffic was an HTTP 
Get 
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Why even have AppIDs/Fingerprints? 







What’s the point of AppIDs/Fingerprints? 

For one, they give you a powerful tool for 
the quick analysis of what applications are 
being seen in your traffic. 



A simple histogram on AppID allows you to 
quickly identify all of the applications seen 
for a given result set, without needing to 
view each piece of content 
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Why even have AppIDs/Fingerprints? 







Ex: Histogram the applications used during 



Target activity: 




[HEstogram Grid *1 
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Why even have AppIDs/Fingerprints? 







Secondly, they provide an additional criteria 
that you can use in your query. 



NOTE: It’s important to point out that 
since most AppIDs + Fingerprints are 
tagging technology and/or applications, 
they SHOULD NOT be the sole criteria 
for your queries in X-KEYSCORE! 
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Why even have AppIDs/Fingerprints? 




EX: I’m looking for targets using mail.ru 
from behind a large Iranian proxy: 



IP Address: 





AppID 

( + Firigerprints) [ fullte^t l : 



1 




Frnkl Boilriler 




AppID f Fn d-q 0 ] iso- p q-u ' 1 s]i 


•z 

c 

fa 


mail/webmail/mailru 




m.a i l/web ma i l/rna i Ir u 


rnail/webma il/mai Iru/attach ment L 

mail/webma il/mai Iru/post 
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Why even have AppIDs/Fingerprints? 




EX: I’m looking for targets using mail.ru 
from behind a large Iranian proxy: 



IP Address: 





AppID 

( + Firigerprints) [ fullte^t l : 



1 




Frnkl Boilriler 




AppID f Fn d-q 0 ] iso- p q-u ' 1 s]i 


•z 

c 

fa 


mail/webmail/mailru 




m.a i l/web ma i l/rna i Ir u 


rnail/webma il/mai Iru/attach ment L 

mail/webma il/mai Iru/post 
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Why even have AppIDs/Fingerprints? 



EX: I’m looking for Mojaheden Secrets 2 
use in extremist web forums: 




Field Builder 
AppID C-:Tco-t:ge:p[i j cjiij:sJ 



thru m/extremist/I 



iui li 1 1 Ij'ca u tsi i iisyar icnujd 

for u m /e k tr e m i s t/a I- fir daw s Arab i c 

fbrum/extremist/al-firdawsEnglish 

for u m /e x tr e m i s t/a I- h isba h 

fbrum/extremist/al-hisbahWorkshop 

tbrurn/e k tr e m i s t/a I- i k h las 

fbrum/extremist/al-nukhbah 

for u m /e x tr e rn i s t/a I- nusr ah 

fbrum/extremlst/al-qiirimah 

tor urn /e k tr e m i s t/a I- s hur a 




1 1 

fbrum/extremisf/al-tawhid 


for u m /e h tr e m i s t/a 1 ja z eer ata Ik 




fbrum/extremist/alm3refh 




tor urn /e k tr e m i s t/a m b 




for u m ,/e x tr e m i s t/ash iy a ne 


jiii 



Field Eu i Icier 



AppXD (-FFiTigSuWMSj) 



mo 



J 



e ricr y pt io n/mo jah ede n2 
encryption/mojahederi2/encodedheader 
e ricr y pt io n/mo jah ede n2/h id den 
encryption/mojaheden2./hidden2 
e ncr y pt io n/mo jah ede n2/key id s 
encryption/mo jaheden2/secu refile 
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in order to assign the AppID tag. 

Example, let’s say that this is the definition 
for mail/webmail/yahoo: 

appid ( ' mail/webmail/yahoo ' , 9.0) = 'Host: mail . yahoo ' ; 
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r 

Here is a client side Yahoo session: 




GET /login. html HTTP/1.1 

Referer : http : //us . f35 9 .mail .yahoo . com/ym/ShowLetter 
Accept-Language : ar 
Accept-Encoding : gzip, deflate 

User-Agent: Mozilla/4 . 0 (compatible; MSIE 6.0; Windows NT 5.1; SVl) 
Host: mail.yahoo.com 
Connection: Keep-Alive 

Cookie: B=f n50ehd2612o2 &b=3&s=rp ; YMBM=d=&v=l; 
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appid ( ' mail/webmail /yahoo ' , 9.0) = ' Host : mail . yahoo ' ; 



GET /login. html HTTP/1.1 

Referer : http : //us . f35 9 .mail .yahoo . com/ym/ShowLetter 
Accept-Language : ar 
Accept-Encoding : gzip, deflate 

User-Agent: Mozilla/4 . 0 (compatible; MSIE 6.0; Windows NT 5.1; SVl 
Host: mail.yahoo.com 
Connection: Keep-Alive 

Cookie: B=fn50ehd2612o2&b=3&s=rp; YMBM=d=&v=l; 



Application: mail/webmail/yahoo 
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How AppIDs work 



What does the number in the AppID mean? 
a p p i d m a i I/we b rna il/yahoo , 9.0)= 

Each session can have only one AppID 

The goal is for the AppID to be as descriptive as 
possible 

Any given session might qualify under multiple 
AppIDs definitions, but only the most specific 
AppID that applies to the session is assigned 

Lowest number wins, so the lower the number, 
the more specific the AppID definition 
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appid for mail/webmail/yahoo/login: 



appid ( ' mail/webmail/yahoo/login , 8.0) = ' Host : mail . yahoo ' and 
' /login 1 ; 

It has a lower number than 
mail/webmail/yahoo, so if it “hits” it will be 
applied 
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appid { ' mail/webmail /yahoo ' , 9.0) — ' Host : mail . yahoo ' ; 

appid ( 'mail/webmail /yahoo/ login, 8.0) = 'Host: mail. yahoo' and 



' /login 1 ; 



GET /login. html HTTP/1.1 

Referer : http : //us . f35 9 .mail .yahoo . com/ym/ShowLetter 
Accept-Language : ar 
Accept-Encoding : gzip, deflate 

User-Agent: Mozilla/4 . 0 (compatible; MSIE 6.0; Windows NT 5.1; SVl 
Host: mail . yahoo . com 
Connection: Keep-Alive 

Cookie: B=fn50ehd2612o2&b=3&s=rp; YMBM=d=&v=l; 



Application: mail/webmail/yahoo/login 
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Note that the AppIDs have a directory-like 
structure: 



mail/webmail/yahoo and 
mail/webmail/yahoo/login 

If you wanted to search for all webmail activity 
you could search for mail/webmail/* 

If you wanted to search for all Yahoo mail 
activity you could search for 
mail/webmail/yahoo/* 

etc 
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How AppIDs work 



Some session can hit on many AppIDs. 
For example a single session might hit on: 




appid('http/response', 9.2) 
appid('mail/webmair, 8.9) 



appid('mail/webmail/yahoo\ 6.0) 
appid('mail/webmail/yahoo/attachment\ 5.0) 



Which one will be assigned as the winning 
AppID? 
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How AppIDs work 



When you see an AppID how do you know 
what was used to define that AppID? 




Through the XKS AppID signature page 
available through “go xkeyscore” 

Or by simply clicking on the hyperlink 
AppID from the new GUI! 
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What is a fingerprint? 



AppIDs were built to describe applications, 
of which there *should* only be one 
application seen per session. 




How do we describe other attributes of a 
session that aren’t necessarily tied to a 
particular application? 



TOP SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZL 




A particular type of encryption could be 
used in Yahoo Email, Gmail Email, SMTP 
Email. 



It could be used inside of a Word 
Document being uploaded to a free file 
website. 

It could be used inside of a private 
message sent through Facebook. 

Etc. 
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of encryption regardless of the application 
we saw it in? 

Answer- Fingerprints 

Think of Fingerprints as “attributes” of a 
session. 

A session can have as many fingerprints as 
is needed to best describe it. 
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appid { ' mail/webmail /yahoo ' , 9.0) = ' Host : mail . yahoo ' ; 

appid ( 'mail /yahoo /login , 8.0) = 'Host: mail. yahoo' and ’/login 1 ; 



fingerprint ( 'mail/arabic ' ) = 1 mail 1 and / language [ : =] ?ar/; 



GET /login. html HTTP/1.1 

Referer : http : //us . f35 9 .mail .yahoo . com/ym/ShowLetter 
Accept-Language : ar 
Accept-Encoding : gzip, deflate 

User-Agent: Mozilla/4 . 0 (compatible; MSIE 6.0; Windows NT 5.1; SVl 
Host: mail . yahoo . com 
Connection: Keep-Alive 

Cookie: B=f n50ehd2612o2 &b=3&s=rp ; !MBM=d=&v=l 



Application: mail/webmail/yahoo/login 
Fingerprint: mail/webmail/yahoo/login mail/arabic 
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Appid vs Fingerprint 

J. I 1 U 1 0 1 0 1C f ' * ® ■ 




Each session gets one appid -- lowest level wins. It gets databased in 
the 'application' field. 



All matching fingerprints are stored in the 'fingerprint' field. 



Application Type; 




V 




Application Info: 




Winning appid 


Application: 




vjr Winning appid + 






all fingerprints 


AppID 




— TPopulate with Field Builder! 


f + Finqerprints'} f full text 1 : 




[Populate with Tree Field Builder! 
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Fingerprint Examples 



Ex: E-Mails mitb @Gi)<grwpti®n 




From: "La unchpad OpenPGP Key Confirmation" <noreply@launchpad.net> [Save Address] [Block Sender! 



To: : 

Cc: 

Subject: Launchpad: Confirm your OpenPGP Key 
Date: Wed, 31 Dec 2006 10:04:16 -0600 



BEGIN PGP MESSAGE 



varsinr 

A nrdi r^sti nn 


vi 4 n il-jnuxi imixi 
Ar^riir3 f-i-fi nn^rririntc^ 




/A|J|JII LdU Ul 1 


1 MLJfcl |JI 1 1 ILb J 




mai l/webmail/outblaze 


rnai l/webmai l/outblaze ha s_fi ngerpri nt encrypti on/p g p encrypti on/ p gp/mes sage 





spflvtVPZsIl vpg67VdH F U p rgvOJ p mj Q I b73 gWmh b OUrZzy G dDRIaS C cF zJA7 0 1 L 
3Xy C rlri i n i J4/c9B +k h D az h 1 X Y/S7y N i33Wrl kd3 GO z9DF F 1 1 N u3 1 nwjh3 +n cO p v 
Oly zts GzLFBJS+qJrPvm KBfzz7tWp2dj x yfM GoA Y NAf/GO o hRO Bj qTg O U I qLRVrE 
eEFivrM O nBxCOSHIF ra7 LpZI sTU Fp BJ NAkgg u k7 mBfJO dM m UQVSlYlelYl 1 xS GuWv5 + 

U k4 b BwwZI Vp E VH C y G uv8 ux- +V+KpSk QtDwd h I pi 2 SZ2S U rnl u pnVBS Ifcnl hVWxZp 
La Y3 mXq N W hyhzFPFxkh U wq z d/ rM x r C J u c fXG aeisSizZDIQO WxTS we7 B wvG8 Bvrir 
QEQVK Y30vW'g +2 pDTPrKq3u E q OwjOJ Y7 KTPlYl 12 gZLNABD u C J m5 1 RALZq qETTg4 d h 
xVO r9+2ZLty G DXQ h LM yBElYn s4+j i P 1 rd3E+TVV7 J VU e/dPI uy 04 DwOUPk Iwu H cC+ 
StLAu GHM S6Rk B4 aDNd i6 GG9kEWvjq2Pvfu M I BWoBjJS R F o D S k 8 q5t 1 ukgeCxrS^r 
Q4 eTm OFTIA7 1 G3 1 2Xa7Zn i Oz v xiWZ4CAb h H LF +3 ba FD3 I b4/E Fm R vP B d qyGwU y HD 
Z5EXyHDzl4XIDyEe/aomEqAsUqPs3MZirHHzpbaS3LbG5B5VKAKU59bENpf/KOgT 
a3 1 UAeGI 16x LzgToVdfh EkPjS b xO D rWcZtHeTEtl n V+3 pc2 P58 +G 1 0 D OETiDCA/j 
dh G2b rU wbx n y6Ap7fU5 e 1 ALU3 ry oXKVtO e CXZH o o Y/p9Q I C3 ko H CVVpt G D6g KC x It 
KW/K5 M +Hk x hHy4 WWb 1 37 C Stz e Ld ai3Bd U43KhOZQWWjK7 pDXKKh H LY IGlawRS c Qa 
eB J +y4 J R 1 KKyXiXY94 E ra a/PO Fz u Y V/G C J U D p qWFR22 bXu y4 Fh k as LWIVI8G +UBHVt 
UfgRx qSssGO D h B D WyCSe LE AdE92TYffJgXOvAOzTqBrP7 uZi/Q7ABFFGTQ9n 
=N4CJ 

— END PGP MESSAGE — 

Thanks, 






Look at the definitions (notice any overlap?): 

fingerprint( l, enc^}/[otion/pgp , ) = 

begin pgp message' or 'begin+pgp+message'; 

fingerprint( E 8ncryption/pgp/message')= 
/(?:BEGIN|EMD) PGP MESSAGE/; 
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Ext Extnmist Forumm private M@ssijg@s 




HTTP Header Information 


Content Type: HTTP/P OST/Form- Data 


POST /vb/private.php?do=insertprri &pmid= HTTP/1.1 





Accept: i mag e/gif, i ma ge/x-x bitmap, image/jpeg, image/pjpeg, applicatior/K-shockwave-flash, application/vnd.ms-exce 

a p p I i c at i □ n/Vn d . m s- p o we rp o i n t , a p p I i c at i o n/m s wo r d , £/* 

R ef e re r: http: //a l-f a I oj a . i nfo/v b/p rivate. php?do=n e wp m &■ u=9B 92 

Accept-Language: en-gb 

Content-Type: applicat ion/K - www-f o rim - u rl encoded 

U A- CPU: x36 

Ac cept-En coding: gzip, deflate 

Us e r-Aci ent : Mo z i 1 1 a/4 . 0 fc □ m p at i b I e: ME I E 7.0 : Wi nd o ws NT 5 . 1 : F D M 



Application 



AppID OFingerprintsi) 




ii^iilyW^I>ii^iil/vl>ulletiii/piivat€jrH^$^ge.iii^ort n i Vhi L- w e l> n i Vki l/uln-i. il I et i i n. | > r iurigjiiessageuiisert hns_fiiiiger print for um/em emisl/al-lftloja 

recipients 
lie c recipients 

Lfljjcj l__\uii_ijQ i i- wi - v 3-i ^2009-01-05.-! I jjq! I _a 1 ^ 3 0 US o y* 1 i ^ i ■ ^ *•- L. 1 1 1 (j ■ i -^i r _ ii j ■ ■ ^ b -3 j . <i n 1 1 i 1 i-r -- i_i i \ ± i nC i uni 

■Ft --I -- 1— 1 1 l>° cloUe , ^ii --. ■■! I i -j i .^j'i iLjloU t ' 1 ih ^jLlcJI y iia- II ^ obsaiJl X jqjQ .LUi< ^ I 4jjiC -J l I ~ I L r Ui( LiujLul r A^ o . Jc. 

LjIjn^ii^S l_q 



title 



message 



LclIIz^. .j i I J JJClI I pl_=Jl ^ jLllJ I l_jjJlLC- La J I jjJ I j J?J I l ' ~ 1 1 ^ - l ~- jl LcuQ -i 'liil n i JjlLte i j-iSl-LLlI «J jj f} 1 Vfr I i'i^I 'i In'! Fj 

J J.- jj \_q j-iizJil jilj - »^l Cl \ _A I f . — ( ^b X .•-. jm-- _;u Liui, ijjjj *- * * ■ <)■ — o i dlL^Aa! I J ■ j * L - -■ ,f 7 I q l=A j - ■ - j A**- 
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AppID vs Fingerprint 

ill 10D1U 10 t! ,yi - > 1 1 ' . 4l IJM 




AppIDs and Fingerprints use the exact same 
language inside of XKS. 



You can tell which one it is by the definition: 



appid (mail/webmail/yahoo) 
fingerprint (encryption/pgp' 



TOP SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZL 



TOP SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZL 



AppID/Fingerprint Language Evolution 










There have been 4 generations of XKS 
AppID/Fingerprint languages 



1 st Generation: Simple Keyword Scanning 

2 nd Generation: Context Aware Keyword 
Scanning 

3 rd Generation: Code based 
AppIDs/Fingerprints 

4 th Generation: Code based AppIDs that can 
extract meta-data (also known as Micro 
Plugins) 
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1 st Generation AppIDs/Fingerprints 




i - 






In the beginning, AppIDs and Fingerprints 
were just keyword scanning similar to 
CADENCE tasking Ex: 



appidCmail/webmail/yahoo', 9.0) = 

'Host: mail.yahoo'; 
appid('mail/yahoo/login, 8.0) = 

‘Host: mail.yahoo' and '/l@giDu'; 
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1 st Generation AppIDs/Fingerprints 







1 st Generation would also support Regular 
Expression (REGEX’s): 



fingerprintCencryption/pgp/message’^ 

H:BeHiI|e|d1 PGP MESSAGE/; 



(instead of quotes REGEX’s are enclosed by 
forward slashes) 
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As well as Hex scanning: 



appid ■database/ms_sql_s@rver(tds)/login , 1 7.5)= 

\xD6\x83\xf2 \xf8\xfdx 0 0\x0 0\x 0 0\x 0 0\xe 0\ 
x03\x00\x00\x88\xfMixffiX36\x04\x00\x00'; 



(Hex characters are prefaced by \x) 
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introduced XKS’s context sensitive scanning 
engine. 

For example, rather than scanning an entire 
session top to bottom to look for 
‘facebook.com’ we can just use the 
dictionary context http_host to target the 
scan for the host field only. 
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in order to assign the AppID tag. 
Example, this is the definition for Hi5 

a p p i d ' m a i l/we b rn a i l/li i 5 r , 6 . 0 1 )= 

tiiSloggedln'c or 
http Jiost(' h i5 . co m : ' o r 
fotml title r hiS : ); 
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) 




O 







If you look at the raw text of this traffic, one 
of the definitions for the mail/webmail/hi5 
will hit: 



'll! 



■ ■ i • 

Header (3) 



a - 

Meta (5) 



Attachments (2) 







< ! D O CTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Trans itional//ECT" "http://ww 
<html xmlns = "http : / /m .w3.org/1999 /xhtml rr > 

<head> 

html_title('hi5'); 



<title>lii5IYourFriend3^Your¥orld^</title^ 

<me t a http - e qui v= " C onten t-Typ e " c onten t = " tex t /h tml ; charset=utf-3" /> 



Registration is quick and easy! 



Rpnistpr 
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— 
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2 nd Generation AppIDs/Fingerprints 



Example: 



Sfacebook = 

html title [ 1 Fscebook 1 ) or: 
http_ho3t [ 1 .facebook.com ) ; 

appid [ 1 social/f acebook 1 , 3.0, webproc= 1 Facebook 1 J — 

$f acebook; 



Note the use of the chain word $facebook in 
the AppID definition 




TOP SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZL 




TOP SECRET//COMINT//RELTO USA, AUS T CAN, GBR, NZL 



% i . I f'jfi I Mi > 1 ' “ 1 d j 

2 nd Generation AppIDs/Fingerprints 



T> * 

^ Urn 



ti 



■ V Uli 



i 



Sfacebook — 



html title ( 1 Faceboofc 1 ) or 
http host ( 1 . faceboofc □ corn 1 ) ; 



app ( 1 social/ f acebook 1 , 3.0, webproc- 1 Facebook 1 ) — 

$f acebook; 



f- 



GET /yovilleA 


de w _gifts|php? gifts liip=l &LI - 1 HTTP/1.1 




Accept: 


image/ gif, image/x-xbitmap, image/jpeg, image ppp eg, application/x-shockwi 


rve -flash 


Accept- 






Language: 






UA-CPU: 


sS6 




Accept- 

Encoding: 


gzip. deflate 










|Host: 


apps.focebook.com 1 




Connection: 


Keep-Alive 




Cookie: 


datr=l 251 060871 -982tl5658affe41 52eSS16a7958b9b9503 Ib60aea9fffaecd04f34 
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2 nd Generation AppIDs/Fingerprints 



/ 



Sfacebook — 



html title [ 1 Facebook 1 J or 
http host ( 1 . facebook. com 1 ) ; 



T» ■■ 

Hjl 



s 



ti 



■ I LfU 



■ 



irte 

f - 






appid ( 1 social/f acebook 1 , 3.0, webproc— 1 Facebook 1 J - 

acebook; 



All of these hosts 
would match this 
AppID: 



7 



Host 

platform.ak.facebook.com 

vtriuinb.iik.fiicehook.com 

creative.nk.facebook.com 

www.facehook.com 

02959290782.chaniiel32.facehook.com 




apps.facebook.com 

facehook.com 

03458988995.cliaiiiiel32.facehook.com 
static.ak.facehook.com 
h.st atic.ak.facehook.com 
O3881417OOO.chamiel32.facebo0k.com 
liadge.facohook.com 
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2 nd Generation AppIDs/Fingerprints 



mm a 



VC 






■ wu 






Example: 



5 kaspersky_ip - 

ipT'SO. 

ip ( '60. 
ip ( 1 80 D 
ip ( 1 80 . 
ip ( 1 80 = 
ip ( '80. 
ip ( '80. 
ip ( '80. 



239 

239 

239 

239 

239 

239 

239 

239 



144 

144 

144 

144 

144 

144 

144 

144 



72 

73 

74 

75 

76 

77 

78 

79 



J or 
J or 

) or 
} or 
) or 
) or 
) or 




appid ( 1 antivirus/kaspersky 1 , 1.0) - 

Skaspersky ip; 



appid ( 1 antivirus/kaspersky/updater 1 f 5.0) 

port (21) and $kaspersky ip; 
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2 nd Generation AppIDs/Fingerprints 



Can you tell what’s going on here? 



appid [ 'Hiail/iAjebmail/netlog 1 , 8.0, viebproc- 1 Metlog 1 ) — 

html title [ 1 Netlog 1 c) or 




http_host ( 1 . netlog. com 1 ) or 

http cookie ( /domain= . { 3 , 10 } \ . netlog\ . com/ ) ; 
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Mobile User Agent fingerprints: 



fingerprint [ 1 browser /cellphone/iphone 1 ) — 

browser ( 1 iPhone 1 J ; 



fingerprint ( 'browser /cellphone /motorola 1 ) - 

browser [ 'MOT- 'c or 'motorola' J ; 



fingerprint ( 'browser/ cellphone/sony_ericsson 1 ) - 

browser ( 1 SonyErricsson 1 J ; 

f ingerprint ( 'browser/cellphone/blackberry') = 

browser ( 1 BlackBerry ' J ; 
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USSID18 Considerations! 



r r _ 

If you were to query on any of these 
fingerprints by themselves, would your 
auditor be happy? 



fingerprint ( 1 browser /ceiiphone/iphone 1 ) — 

browser ( 1 iPhone 1 ) ; 

I fingerprint ( 1 browser/cellphone/motor ola 1 ) = 

I browser ( "MOT- 1 c or 'motorola'); 

fingerprint [ , browser/cellphone/sony_ericsson ' ) = 

browser ( ' SonyErricsson 1 ) ; 

■ fingerprint ( 'browser /cellphone/blackberry ' ) = 

■ browser (' BlackEerry ') ; 
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But if you were to query on an Afghan IP 
address that was a valid foreign intel target, 
and then “AND” it with those fingerprints, 
that would be a USSID1 8 compliant query 
(and your auditor would be happy) 
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3 rd Generation AppIDs/Fingerprints 



V 4 

3 rd Generation AppIDs/Fingerprints 
introduced the ability to have code-based 
scanning 




Why is this important? Because scanning 
sessions for keywords, hex values and 
regular expression can only take you so far. 

Using Code-based AppIDs, we can run 
statistical tests of the data that can help 
determine what type of data it is when 
keyword scanning can’t give us a result. 
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4 th Generation AppIDs/Fingerprints 



4th Generation AppIDs/Fingerprints 
introduce the ability to extract and database 
meta-data from Appid/Fingerprints 

Why is this important? 

With the dynamic nature of DNI applications 
we need the ability to quickly react and 
deploy solutions to extract new fields of 
meta-data that are important to analysts 
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4 th Generation AppIDs/Fingerprints 







Previously, if we identified a new protocol or 
a new field that we wanted to extract meta- 



data, we would need to upgrade a “core” 
plug-in and wait until we could upgrade the 
field sites. 

With 130 field sites, each on their own 
upgrade schedule, this could take months for 
a simple change to get out in the field 
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_ j _ , — . . . U- -V - 1 I tj 1 I 1 i 'I |l m ± _ , 

* l . It*.' ' A • ’ 

4 th Generation AppIDs/Fingerprints 



With 4 th generation AppIDs, a new protocol, 
meta-data value, can be properly processed 
within an hour of updating the 
AppID/Fingerprint. 
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4 th Generation AppIDs/Fingerprints 



■ 






r i 




» I wu 



j4 







Examples: 



appid [ 



social/f acebook/chat/toserver 1 f 1.0) = 
http_host ( 1 fac ebook. coin 1 ) and 
$http post and 
url ( 1 /aj ax/chat /send. php 1 ) 

: C++ 

extractors = { { 

lo gin_email — /login_x— . * ( [ a-s0-9_\ . ] {30 } %40 [ a-s0-9_\-\ . ] {3 0} ) / ; 
text — /msg_text= [ [ A £\n\ r ] +) /; 

}} 

main = { { 

if ( ±o gin_email ) { 

xks : : user_activity_t ua ( "chat ", rr facebook rr ) ; 

ua . client . add ( xks : :urldecode [iogin_email[0] ) , "facebook") ; 

ua. apply (J ; 



if (textj { 

xks : : chat body ( xks : : urldecode [ text [0 ] ) ) ; 



return true; 

}}; 
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Example 



Let s take a closer look: 



First a V4 AppID needs to be “anchored”. 
The anchor is the beginning part of the 
AppID 



appid ( 1 social/f acebook/chat/to server', 1.0) = 

http_host ( 1 f acebook. com 1 ) and 

$http post and 

url [ 1 /a j ax/ chat/ send . php 1 ) 
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ft. I 4 i s M f l i J 1 rl 

Facebook Chat V4 Appid Example 




DNI Presenter Display: 



Session Header [3) Attachments (3) Mete (9) 




I TS Web Form Display 



Form Fields 



msg_id 
client time 




1 250 6-4 2 1 S 0 3-4 2 



to 

num_tabs 
pvs_time 
msg test 
post_form_id 
fb_d.tsg 




1 



1250642145719 
dont u stiU recognize me? 

ecba32 6 db 1 dO 5 04 97f8a 1 8£8 9 24£k8fd 
GMFF 9 1 S"W7C8 AX_L7ID - kiN7 cL 3 8 E 
p o s t_fb rm_id_s o urc e A syncF.e que st 
a 1 

n|trjd] c 345511 63d43Sfb 1 ec7c5a5430£h9432 

nctr[nid] 4 6fc ef7f8c 1 f23 6§4 d 1 e 024 6c 2d7 34 a0 

ifjct] 1250642184720 
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Facebook Chat V4 Appid Example 





Lets look at the raw: 



Session 


Header (3) Attachments (3) Meta (9) 




ASCII ^ t . Enter text to searc 


» 


POST http: //wm. facebook. com/ aj ax /chat/send, php HTTP/1.1 
Host: WM. f acebook . com 

User-Agent: Uosilla/5.0 [Windows; U; Windows HT 5.1; en-ITS; rv: 1.9. 0.13) &ecko/2009073022 Firef ox/3 . 0. 13 
Accept: text/html^application/xhtml+xml , application/xml ;q=Q.9 i rV ff ;q;=Q.B 
Accept-Language : en-us , en; g=0 . 5 
Accept-Encoding: gsip^def late 

Accept-ChaEset: ISO-8859-l.rUtf-B ;q=0.7, r ' ?, ';q=0.7 
Keep-Alive: 300 
Proxy- Connection: keep-alive 
X-3VW-Rev: 181721 

Con tent -Type: applicaticin/x-www-f orm-url encoded; charset=UTF-3 
Referer : http : //ww. facebook. com/editpictuEe .php ?succes3=l 
Content- Length: 366 

Cookie: datr=1243211999-a94dd86bll6554d2b5£ d0148010051.b7e71.6b386c627c920a4e03; s_vsn_f acebookpoc_l=164069410^ 
Pragma: no -cache 
Cache-ContEol : no-cache 

m3 g_i c 1 i en t_t ime =12506 42 1 8 0 3 42 & to ab 3=1 &pvs_ time =1250642145719 sms g_ tex t= dont% 2 Qt 
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>10 01 



1001 1 001 tool 

1001 1001 1001 
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Facebook Chat V4 Appid Example 




The “anchor” of this V4 AppID was present: 



appid ( 1 soc±al/f acebook/chat/to server 1 f 

http_host [ 1 f acebook.. com 1 ) and 

$http post and 

url [ 1 /a j ax/chat/send . php 1 ) 



1 . 0 ) = 



» HTTP/1.1 

| Host: ww . facebouk. com | 

User-Agent: Hoz i 1 la/ 5 . 0 [Windows ; U ; Windows HT 5.1; en-US ; rv : 1 . 9 . 0 . 1 3 ) 
Accept: text/html , application/xhtial+xml , application/xml ;q=0.9,*/*;q=0-3 
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Facebook Chat V4 Appid Example 



Once the “anchor” hits, the rest of the code 
executes. In this case, we’re looking for 
these two REGEX’s from the “Extractors” 
section: 



extractors - { { 

login_email = /log±n_x= . * ( [ a- z0-9_\ -\ . ] { 3 0 } *40 [ a- z0-9_\ . ] { 3 0 } ) / 

text = /msg_text= ( [ A &\n\r ] +) /; 

}} 
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TOP SECRET//COMINT//RELTO USA, AUS, CAN, GBR, NZL 



Facebook Chat V4 Appid Example 





This REGEX hits within the large cookie string 



login email = /login . * ( [a-sO-9 \-\ . ] {30} §40 [a-sO-9 \-\.]{30})/; 







login_email = /login_x= . * ( [a-z0-9_\-\ . ] {30} *40 [a-a0-9_\-\ . ] {30})/; 





= a % 3 A2 % 3 A% 7 B s % 3A5%3A%22emai 
I % 2 2 % 3 B s % 3 A2 6 % 3 A % 2 2 
| yahoo.com%22%3Bs%3A1 9%3A%22 
remember_me_default%22%3Bb%3A1%3B 
%7D; 
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Facebook Chat V4 Appid Example 



The other REGEX: 








u i ISO 

vy 



n 









o 



dont%20u%20still%20recognize% 



20me%3F post_form_id 
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Facebook Chat V4 Appid Example 



Finally, in the “Main” section, if those 
REGEX’s found the data they were looking 
for, they get databased 



main = { { 

if (login_email) { 

xks : : user_activity_t ua ( "chat ", "facebook") ; 

ua. client . add [xks : : ur idee ode (login_email[0] ) , "facebook") 

ua. apply ( ) ; 

} 

if (text) { 

xks : : chat_body ( xks : : urldecode ( text [ 0 ] ) ) ; 

} 

return true; 
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4 th Generation AppIDs/Fingerprints 



. * 



If 

FI r 



i 



r n 

■ 




crscoRu 

4u 




Another example: 



&p p id ( 1 f lie t r ans f e r /we b / z s ha re.net / up 1 o ad/ response , 5.0) — 
http title f 1 z SHARE 1 ) and 1 zshare . net /delete . htrnl 1 

: C++ 

extractors : { { 

w±t_f±le_ name = /The'\ s f lie \ s ■=: s t r o n gxf o nt \ s c o io r= \ rr #33 3 3 3 3 \ rr > [ [ fS < ] { l r 30Q} ) \s </ ; 
wf t delete ur 1 = / s shar e . ne t \ /delete . htreil\ T ( [ 0- 9 ] + 1 = ( [ 0-9 a - zA- Z ] {32} ) \ rr / - 

wf t _ up lo a d_id — / < f o sit c olo r— V p # 6 6 6 6 6 6\ PF >< a hr ef=\ " h ttp: \ / \ /www\ . z s har e \ . ne t \ / [ A \ / ] + \ / I [ A \ / ] + j / ; 

r.-.if t url = / <f o nt co lo r= "#666666 S p p Xa hr e f = \ " (http: \ / \ /w»i \ . z s h ar a \ . ne t \ / [ % / ] +\ / [ \ / ] + ) / ; 

wf t _ up lo a de r_u ser name = / <s ma 11>L o gge d in as: ( [ A < ] + ) < \ / s rn all>/ ; 

}} 

main = { { 

if n.-jf t delete url ) { 

DB [ "web file transfer"] [ "wf t upload id"] = wft upload id[0]; 

DB [ "web file transf er rr ] [ rr wf t delete rr ] = wft delete ur 1 [0] + rr - "+wf t delete url[l]; 

D B [ " w eb_f i le_ t r ans f e r ,r ] [ F "wf t _ s it e _ name " ] = " s s har e . ne t r f ; 

D B [ FF w ekif i le_ t r a ns f e r " ! ] [ p p t r an s f e r _ t y p e pp ] = FP up lo ad"; 

if (wft file name) { 

D B [ "web _ f lie _ t r ans f e r rr ] [ rr wf t_f 11 e name r r ] = wf t_f ile_n ame [ 0 ] ; 

} 

if (wft url) { 

DB [ "web _ f lie _ t r ans f e r " ] [ "wf t_url pp ] = wf t_url [ 0 ] ; 

} 

if ( v.) f t _up lo ade r _us e r name ) { 

D E [ "web _ f lie _ t r ans f e r pp ] [ rp up I o ade r_us e r name " ] = wf t_u p lo ad e r _us e r nain e [ □ ] ; 

} 

DB. apply () ; 

} else { 

logger, deb u g ( rr f lie t r a nsf e r / we b / s s ha r e . ne t / up 1 o ad/ res p o ns e 1 H o s fc re ge x s didn 1 1 raa t c h rr ) ; 

} 

return true; 

}}; 
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FFU Successful Upload Pages 
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Welcome to Z SHARE 

With zSHARE you can upload files, images, videos, audio and flash for free. Simply use the upload form below and start sharing! You can also use 
zSHARE as your personal file storage: backup your data and protect your files. First Time? Read our FAQ ! 

* Upload now 

* Login 

* Create Free A ccount 

* Premium 

* FAQ 

File Uploaded 

The file wok.rm was successfully uploaded! (1S.4SMB-). You're now ready to share it with unlimited people or keep it as a backup. 



Download Link 



http ;//www, zs h are . n eh do wnlo a 4/64 3 '6 '34 j 6 2 1 fO 3 5 b 1 / 



Link for forums : 
Direct Link: 
Delete Link: 



[U RL= http ://www i s h are . n et/d own I □ e. d /6 4 3 8 3 A 5 6 2 1 f 0 8 5 



http ://www. z s h are . n et/d own I □ ad/G 43S345621fO0561/ 



http ://www. zsh are . n et/d e I e te . htm I ? 6 A 3 3 345 6-7 7663 9 3 5 e 
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FFU Successful Upload Pages 



o ) 0 | 01 I 

> ioi fltooo! 
0 cn toioir 



Again look for the anchor to hit in the raw traffic 



appid ( 1 file transfer /web/ sshare . net/ upload/ response 1 , 5.0) 

http title ( 1 z SHARE 1 ) and 1 zshare . net/ delete . htiril 1 



<title>zSHAEE - Free File, Image and Video Hosting</title> 



value = "http : //ww. zshare.net/delete.htDiLl?i 
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FFU Successful Upload Pages 





Next look for the extractor REGEX’s to match 



extractors 



wft file name = /The\sf ile\s<strong><f ont\scolor=\ "#333333\ "> ( [ A <] { 1, 300} ) \s</; 



class= rr textl rr >The file <strong><f ont color= rr ^333333 rr >wok. rm </font></stroncj> 



Then database what was extracted 



hi a m - { { 

if ^ft file name J { 

DE [ "web file transfer" ][ rr ™ft filename"] 



— ■wft file name[0]; 
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